1. Field of the Invention
This invention relates to a speech interface in a home network environment that can dynamically and actively extend its vocabulary on basis of device or medium dependent vocabulary transmitted by network devices. In particular it relates to a method of extending the vocabulary within a speech unit realizing such a speech interface.
In this context, a device can either be hardware, e.g. a VCR, or software, e.g. an electronic programming guide.
2. Description of the Related art
The EP-A-97 118 470 describes that in a home network environment a device can send its vocabulary describing its functionality and its speech interface to a speech unit. The speech unit can then translate a received and recognized user utterance into a corresponding user-network-command to control said network device on basis of the received vocabulary.
In September 1998 Motorola published a language reference to VoxML 1.0 which is a language based on XML. This language is used for describing dialog systems by specifying dialog steps that consist of prompts and a list of possible options in general. Basically, VoxML offers the same ease of production to voice applications as HTML does as a vehicle for the presentation of rich text as well as images, hypertext links and simple GUI input controls. The VoxML language reference contains information on the syntax of the elements and their attributes, example usage, the structure of VoxML documents or dialogs and pointers to other reference documentation that may be helpful when developing applications using VoxML.
Similarly, the paper xe2x80x9ca markup language for text-to-speech synthesisxe2x80x9d by Richard Sproat et al. ESCA. Eurospeech 97, Rhodes, Greece, ISSN 1018-4074, page 1774 discloses a spoken text markup language (STML) to provide some knowledge of text structure to text-to-speech (TTS) synthesizers. The STML text can e.g. set the language and default speaker for that language in a multi-lingual TTS system so that appropriate language and speaker specific tables can be loaded.
Furtheron, Phillips has developed a dialog description language called HDDL specifically for the kind of dialogs that are encountered in automatic enquiry systems. HDDL is used to build a dialog system in off-line mode before it is sold.
The object underlying the present invention is to provide a method to transmit the functionality and speech interface of a network device to the speech unit of the network and to handle such functionalities and speech interfaces of several network devices within the speech unit.
Therefore, the present invention provides an easy, quick and flexible method to control a network device connected to a network with a speech unit within said network that translates user-commands into user-network-commands to control said network device via said network on basis of the functionality and a speech interface that comes e.g. with said network device.
The inventive method is defined in independent claim 1, preferred embodiments thereof are respectively defined in the following dependent claims 2 to 24.
According to the present invention, every network device connected to a network is associated with at least one device-document that defines the functionality and speech interface of said network device. Such a device-document can come e.g. without said network device and comprises one or several pairs of an user-command interpretation element and an associated user-network-command. Furtheron, the speech unit is able to receive such device-documents and merge them together into one speech interface description. This speech interface description comprises then the language of said speech unit. It can also be referred to as a general document for the whole network. The spoken user-commands that are received and recognized by the speech unit are then translated into user-network-commands on basis of all pairs of an user-command interpretation element and an associated user-network-command included in said general document. Such an user-command interpretation element can e.g. comprise a vocabulary element, a definition of a grammar, a pronunciation definition or several of these or other examples.
As described in EP-A-97 118 470, documents for devices might alternatively be fetched by the speech unit from its own memory or a (distant) database.
The adapting of the general document within the speech unit can be done purely syntactically at runtime after a device-document is received from a device. Of course, it is also possible that a network device comprises several documents that individually describe a part of its functionality and a part of its speech interface and only those documents are transferred to the speech unit that are really needed, e.g. only those documents describing the functionality of a network device for a certain language or only documents that define a part of the functionality of the network device. If the speech unit or an individual network device itself then recognizes that further speech capabilities of said individual device are needed, e.g. an additional language, the corresponding device-document(s) can be send to the speech unit that adapts its general document on basis of this further document and can generate the corresponding user-network-commands on basis of the adapted general document at run-time.
The present invention and its advantages will be better understood from the following description of exemplary embodiments thereof. These subsequent discussed embodiments are described on basis of a case where two devices are connected to a network that also includes one speech unit. Initially, the general document of the speech unit is empty and both device-documents each of which defines the language of one device are merged together into one interface description within the speech unit. Obviously, these examples can be extended to a case with n devices being connected to a network or also for the case that the speech unit comprises a general document that gets adapted on basis of a newly received device-document. Furtheron, for the sake of simplicity, a device document comprises user-command interpretation elements consisting only of one vocabulary-element.
In the following examples L1 is the accepted language, i.e. the vocabulary-elements and associated commands, of a first device 1 and L2 that of a second device 2. Mathematically speaking, L1 is a set of at least one vocabulary-element, i.e. word wi (although a word wi needs not to be a single word, it could also be a complete utterance consisting of several words), and the associated user-network-command. Additionally to the vocabulary-elements for example elements about pronunciation, grammar for word sequences and/or rules for speech understanding and dialog can be contained in the language.
In a first example L1 and L2 contain no same vocabulary-elements, i.e. no same words, so that L1∩L2={ }, the merged accepted language L for the interface description is L=L1∪L2. i.e. the general document within the speech unit is built by adding the pairs of vocabulary-elements and associated commands from document 1 comprising the language L1 of the first device 1 and document 2 comprising the language L2 of the second device 2. As the vocabulary-elements together with the associated commands implicitly define for which device they were meant, since L1 and L2 contain no same vocabulary-elements, user network commands can be generated and sent to the correct corresponding devices.
In this first example the two network devices might be a TV set and a CD player. In this case L1 associated with the TV set and L2 associated with the CD player respectively comprise the following vocabulary-elements within the device-documents:
L1={MTV, CNN}, and
L2={play, stop}.
Since L1 and L2 do not contain the same vocabulary-elements, i.e. L1∩L2={ }, the merged accepted language L of the interface description is L=L1∪L2={MTV, CNN, play, stop}. For example, these vocabulary-elements are respectively associated with a user-network-command having a functionality as proposed in the following list:
MTVxe2x86x92 less than switching TV to MTV greater than ,
CNNxe2x86x92 less than switching TV to CNN greater than ,
playxe2x86x92 less than put CD into play mode greater than , and
stopxe2x86x92 less than put CD into stop mode greater than .
However, if the two devices share similar vocabulary-elements, i.e. same words, so that L1∩L2xe2x89xa0{ }, according to the present invention, they are made distinguishable. In a first preferred embodiment of the present invention the device name will be prefixed and/or appended to at least those vocabulary-elements that build the intersection in between the respective accepted languages, i.e. in between L1 and L2. Therefore, the user has to prefix and/or append the spoken user-command within the name of the respective device he/she wants to address. As mentioned above, if the command is not confusable, the device name need not to be prefixed, but may be.
In the following second example a respective command will be prefixed with the name of the device and the new language L that builds the interface description is the union of the non-confusable words of both languages and the languages of the devices prefixed by their device names (respectively called n1 and n2 in the following formula):
L=L1 (L1∩L2)∪L2 (L1∩L2)∪n1L1∪n2L2
The following second example will clarify the inventive method according to this embodiment with the help of a network comprising a CD player and a tapedeck, respectively named xe2x80x9cCDxe2x80x9d and xe2x80x9cTapexe2x80x9d. The accepted language L1 of the CD player and the accepted language L2 of the tapedeck respectively comprise the following vocabulary-elements:
L1={Play, Stop, Skip}, and
L2={Play, Record, Stop, Rewind}.
The accepted language L of the speech interface is L=(L1 (L1∩L2))∪(L2 (L1∩L2))∩n1 L1∩n2 L2={Skip, Record, Rewind, CD Play, CD Stop, CD Skip, Tape Play, Tape Record, Tape Stop, Tape Rewind}. The functionality of the associated user-network-commands could be:
Skipxe2x86x92 less than skip to next track on CD greater than ,
Recordxe2x86x92 less than go to record mode for tape greater than ,
Rewindxe2x86x92 less than rewind the tape greater than ,
CD Playxe2x86x92 less than go to play mode for CD greater than ,
CD Stopxe2x86x92 less than stop playing the CD greater than ,
CD Skipxe2x86x92 less than skip to next track on CD greater than ,
Tape Playxe2x86x92 less than go to play mode for tape greater than ,
Tape Recordxe2x86x92 less than go to record mode for tape greater than ,
Tape Stopxe2x86x92 less than stop to play/record the tape greater than , and
Tape Rewindxe2x86x92 less than rewind the tape greater than .
According to a second preferred embodiment of the present invention, the xe2x80x9csame words problemxe2x80x9d is solved by automatically sending a query to the user to clarify the device the user wants to address in case of a recognized ambiguous command. Formally, the accepted language of such a scenario is the same as with the disjunct languages in the first example, but the interpretation changes. A third example that is used to clarify this embodiment is based on the scenario given in the second example, i.e. a CD player and a tapedeck as network devices. The accepted language L is build as in the first example, i.e. L=L1∪L2, but under the condition that a selection procedure is performed when recognizing a vocabulary-element that is included in the intersection of the languages L1 and L2. Under the condition that L1 and L2 respectively comprise the same vocabulary-elements as in the second example, the accepted language L of the speech unit comprises the following vocabulary-elements: L={Play, Stop, Skip, Record, Rewind}. The functionalities of the associated user-network-commands in this case are:
Playxe2x86x92 less than clarify 1 greater than ,
Stopxe2x86x92 less than clarify 2 greater than ,
Skipxe2x86x92 less than skip to next track on CD greater than ,
Recordxe2x86x92 less than go to record mode for tape greater than , and
Rewindxe2x86x92 less than rewind the tape greater than ,
where xe2x80x9cclarify 1xe2x80x9d means that the speech recognizer outputs the query xe2x80x9con what device?xe2x80x9d and switch the language to La with vocabulary-elements La={CD, Tape} and the functionalities of the associated user-network-commands in this case are:
CDxe2x86x92 less than go to play mode for CD greater than  and less than switch back to L greater than , and
Tapexe2x86x92 less than go to play mode for tape greater than  and less than switch back to L greater than , and
where xe2x80x9cclarify 2xe2x80x9d means that the speech unit outputs the query xe2x80x9con what device?xe2x80x9d and switch the language to Lb with the vocabulary-elements Lb=La={CD, tape} and the functionalities of the associated user-network-commands in this case are:
CDxe2x86x92 less than stop playing the CD greater than  and less than switch back to L greater than , and
Tapexe2x86x92 less than stop play/record the tape greater than  and less than switch back to L greater than .
According to a third embodiment of the present invention, the second embodiment can be accelerated when the speech unit just takes the most recently used device as default device and therefore translates a received and recognized spoken user-command into a corresponding user-network-command on the basis of the general document comprising the accepted language L of said speech unit and the default device. Of course, the possibility is given to put another device into focus again, e.g. by just uttering the name of the device.
According to a fourth embodiment an adaptation of dialog is performed. In the situation as described above, clarifying questions are output to the user to correctly assign ambiguous words and the system observes the reaction of the user. If most of the clarifications yield that the most recently addressed device was meant, the question will be skipped and the user-network-command gets directly issued to that device. Several information sources could be used for predicting for which device a command was given, e.g. not only considering the most recently used device, but also similarities of devices or device classes. For instance, it is more likely that a xe2x80x9crecord thatxe2x80x9d command while watching TV is meant for the VCR rather than for an audio tape, while it is the other way round while listening to the radio. It is also possible that device classes could be learned from the user""s behaviour or by investigating a functional description of the devices.
According to the present invention, i.e. in all embodiments described above, a device-document can be sent from a device to the speech unit directly after said device was connected to the network or after said device received a user-network-command from the speech unit. It is also possible that the device sends such a device document dependent from its current status to the speech unit. A device can comprise more then one document or devices might change their functionality dynamically based on changing content information and might therefore update the dialog and speech understanding capabilities when they observe a change or regularly by sending new documents to the speech unit. It is also possible that a device-document is not stored within the corresponding device, but within a device-document supply device, e.g. within an internet-server of the manufacturer of the device. In this case the speech unit might download the device document after it recognizes that said device got connected to the network.
It is for example also possible that a device contains several documents each including the whole accepted language of the whole device in a language of a different country. Therefore, such a device can e.g. first send a device-document for German speakers to the speech unit and after a corresponding command from a user it can send later on a device-document for English language to the speech unit so that the speech unit can translate either German or English user commands into user-network-commands to control the device.
Apart from vocabulary-elements also pronunciation definitions for words used in recognition or synthesis can be included in the elements of device-documents and therefore can be merged straight forward into the general document of the speech unit. When pronunciation definitions are included in a language they can either be solely used for the recognition and synthesis parts for one device or they can be shared for all devices, i.e. a pronunciation introduced by the first device 1 is also valid for the words being uttered for the second device 2 or any other device connected to the network.
Also for synthesis the merging of several device-documents, i.e. the adaptation of the general document in the speech unit is possible. Answers can be given in response to user queries for several devices as well as for a single one. Of course, it must be assured that the synthesis of utterances from several devices is not output concurrently. Some utterances might be caused by an external event, e.g, a warning, based on a supplied priority. They can interrput other utterances and after an Inactivity period of the user the interrupted utterance is re-introduced and the dialog can continue. Managing several streams of dialog in paralell relies on the fact that all streams are active during user inputs and a stream to which the user""s utterance is considered the most likely to match to is continued.
The general document of the speech unit can initially either be empty or it can include a basic set of elements describing interfaces for one or for various devices. In the latter case, a document sent from a device to the speech unit may be underspecified and (only) contain references to such parts of the general document.
For example, it is possible that a default grammar for the wording of utterances is included in the basic set of user-command interpretation elements of the general document while just the dialog grammar for a particular device is specified in the device-document. On the other hand, it is also possible that the speech recognizer contains a full description of a certain functionality in its general document, e.g. a full description of how to control an electronic program guide by spoken dialog. In this case, the dialog grammar and a grammar for the word sequence may contain variables for programm titles, director names, actor names, time and date information and so on and the document sent from the electronic programm guide device to the speech unit just contains the information to fill in these variables.
In another embodiment the device document sent from a device to the speech unit just contains keywords associated with network commands and category identifiers. Based on these category identifiers, the speech unit knows which grammar for the word sequence to take as a specification for possible continuous user utterances. For example, if the keyword is xe2x80x9cVoice of Americaxe2x80x9d and the category is xe2x80x9cRADIO_STATIONxe2x80x9d then the user might be allowed to say xe2x80x9cI want to listen to voice of Americaxe2x80x9d or xe2x80x9cTurn on voice of Americaxe2x80x9d and if the category is xe2x80x9cRECORD_RADIO_STATIONxe2x80x9d, the possible user utterances might be xe2x80x9cPlease record voice of Americaxe2x80x9d.
The concept/value pairs that form the basis for speech understanding, e.g. (TIME, 8:00) that are needed to start an action can be included as grammar of a certain device in its device document that is sent to the speech unit to adapt the general document. The device-document describing the speech interface contains rules which define a mapping between word sequences and concepts and how to fill in the values of variables. Furthermore, it contains rules which define a mapping from a set of concept/value pairs to an action, (e.g. COMMAND, xe2x80x9crecordxe2x80x9d), (TITLE, xe2x80x9cgone with the windxe2x80x9d) and (TIME, 8:00) might indiciate a recording session on a VCR. The dialog may be user-initiated, i.e. the order of the utterances, e.g. xe2x80x9cat 8 o""clockxe2x80x9d and xe2x80x9cthe movie gone with the windxe2x80x9d, and if the information is given in one or more utterances is left to the user. The system can collect the information necessary to start an action based on the grammar and depending on the user-network-commands sent to a respective network-device this device can send a further document comprising additional grammar to the speech unit so that the speech unit might ask back if information is missing or might be able to assign newly recognized user utterances to further user-network-commands.
Apart from the information concerning vocabulary-elements, pronunciation. grammar for word sequences, rules for speech understanding and dialog as user-command interpretation elements, a device-document can also contain the same information, i.e. vocabulary, pronunciation, grammar and so on for multiple languages assigned to the same or equal user-network-command. This offers users to control the device in any of the languages. As mentioned above, this information can also be included in several device-documents of the same device. This allows to specify the interface of a single speech unit being capable of handling multi-lingual spoken input. The dialog grammar can be selected to be language-independent, because the above-mentioned concept/value pairs are language-independent.
As also mentioned above, the document contains a vocabulary either given explicitly or implicitly by the grammar for the spoken words. For each vocabulary-element, i.e. for each word, optionally one or more pronunciations can be given. If it is not given, it can be generated automatically. However, this is error-prone, especially in the case of proper names, words of foreign languages or abbreviations and if special pronunciations (dialectes or foreign accents) are to be captured. The pronunciation builds the basis for creating a word model inside the recognizer given a set of phone or allophone models.
A device-document might also contain standardized information about the device category, e.g. VCR, and/or about its functionality, e.g. it contains a recordable tape, apart from the user-command interpretation elements and the associated commands to make complex commands that involve more than one device possible. For example, if the user has selected a movie playing some days in the future from the electronic programm guide that is displayed on the TV, he might utter xe2x80x9cplease record this moviexe2x80x9d. That means that network commands have to be issued to send the relevant channel, date and time information from the programm guide device to the VCR and a command has to be sent to the VCR to assure that the appropriate recording will be made. A similar case is to copy a movie from one VCR tape to- another. In those cases the speech interface descriptions of the individual devices do normally not define such user utterances. Therefore, the speech unit can adapt its general document on basis of a reasoning part included in a device-document by a reasoning component which provides the speech interface description (including grammar for word sequences and dialog) for this extra-functionality involving more than one device from one device that might be included in one of those complex commands.
Since all information to control the network devices is included in one general document within the speech unit the processing (parsing) is very easy, especially when there are references between the different possible contents of the user-command interpretation elements and/or double information assigned to the same or equal user-network-commands within the general document.
Therefore, according to the present invention network devices that can be controlled via a speech unit included in the network can send a device-document describing its functionality and its speech interface to said speech unit. The speech unit combines those documents to a general document that forms the basis to translate recognized user-commands into user-network-commands with the help of user-command interpretation elements to control the connected network-devices. A device-document might e.g. comprise the vocabulary as user-command interpretation elements and the user-network-commands associated therewith for the corresponding device. Furtheron, pronunciation, grammar for word sequences, rules for speech understanding and dialog can be contained within the user-command interpretation elements of such documents as well as the same information for multiple languages or information for dynamic dialogs in speech understanding. It is possible that one device contains several documents and dynamically sends them to the speech unit in case they are needed. Furtheron, the present invention enables a device to change its functionality dynamically based on changing content, since a network device sends its specifications regarding its speech capabilities to the speech unit while the speech unit is in use. In case a device changes its functionality dynamically, it generates such a device-document dynamically or updates an existing one, e.g. by replacing/inserting station names, and sends this updated device-document to the speech unit.